Introduction to iset's Computational Photography (cp) Framework and Classes
cpCamera is one of a set of classes that allow experimentation with computational photography more easily and compactly than directly programming the isetcam structures they encapsulate (scene, optical iimage (oi), sensor, and imaging pipeline (ip)). Along with wrapper classes cpScene, cpCModule, and cpIP, it provides a programmable interface (cpCamera) that allows users to create "computational cameras" that employ a variety of algorithms for generating scene sequences, deciding how to capture a scene based on a preview, capturing the scene in one or more frames, and then processing those frames into a photo,
Currently, cpCameras are assumed to have a single camera module with a single sensor, but the architecture is designed to allow expansion to multi-module systems.
History: Initial Version: D. Cardinal 12/2020, iset scene support added 1/2021
Block Diagram of "cp" classes
In this tutorial we'll use sub-classes of cpCamera and cpIP that are enhanced to support several different simulation and processing features. cpBurstCamera and cpBurstIP can use a cpScene object to add both camera and object motion to a 3D scene created using ISET3d (which in turn calls pbrt), or add camera motion to any ISET scene or pre-computed image. They can also capture a sequence of images that already have motion incorporated.
Once the initial sensor images are created, cpBurstIP can either use existing ISET ip algorithms to do burst and HDR at the raw sensor level, or employ higher-level image registration and tone mapping algorithms on multiple frames. A main goal of the cp classes is to allow students and researchers to use the default implementations as a jumping off point for projects of various kinds.
Block Diagram of our "Burst" Camera
Initialize our Computational Camera
As our first step we'll create a camera object based on the cpBurstCamera sub-class of cpCamera. The sub-class adds support for simple image registration and tone-mapping to the base cpCamera class. After we create the camera, we equip it with a single camera module that includes an off-the-shelf sensor and some default optics.
% some timing code, just to see how fast we run...
setpref('ISET', 'benchmarkstart', cputime);
setpref('ISET', 'tStart', tic);
% cpBurstCamera is a sub-class of cpCamera that implements simple HDR and Burst
ourCamera = cpBurstCamera();
% We'll use a pre-defined sensor for our Camera Module, and let it use
% default optics for now. We can then assign the module to our camera:
sensor = sensorFromFile('ar0132atSensorRGB');
% Cameras can eventually have more than one module (lens + sensor)
% but for now, we just create one using our sensor
ourCamera.cmodules(1) = cpCModule('sensor', sensor);
Create our Scene
The Cornell Box + Stanford Bunny is part of iset3d, and is set up to help us by labeling the assets in such a way that we can add motion to them as they are rendered by pbrt. You can use the sliders below to control the lighting level (mean luminance) of the rendered scene, as well as the number of rays per pixel pbrt uses and its "film" resolution. We've kept the defaults low to speed up rendering, but for real experiments you may want to increase them:
sceneChoice = "Cornell Box with Bunny"
sceneChoice = "Cornell Box with Bunny"
apertureDiameter = 6% in mm
% getting the 3D scenes requires knowing their location in iset3D, which we
if strcmp(sceneChoice, "Cornell Box with Bunny")
scenePath = 'Cornell_BoxBunnyChart';
sceneName = 'cornell box bunny chart';
elseif strcmp(sceneChoice, "CornellBoxReference")
scenePath = "CornellBoxReference";
sceneName = "CornellBoxReference";
if pbrtLensFile % support for pbrt lens files is hit and miss
pbrtCPScene = cpScene('pbrt', 'scenePath', scenePath, 'sceneName', sceneName, ...
'resolution', [filmResolution filmResolution], ...
'numRays', numRays, 'sceneLuminance', sceneLuminance, ...
'lensFile','dgauss.22deg.6.0mm.json',...
'apertureDiameter', apertureDiameter);
pbrtCPScene = cpScene('pbrt', 'scenePath', scenePath, 'sceneName', sceneName, ...
'resolution', [filmResolution filmResolution], ...
'numRays', numRays, 'sceneLuminance', sceneLuminance);
end
Read 1 materials.
Read 1 textures.
***Scene parsed.
Let's Look at the Camera we've Created
Using a simple UI, we can control some aspects of our camera graphically.
cpCameraWindow(ourCamera, pbrtCPScene);
Set an Object in Motion in our PBRT Scene
We can set any named object in the asset tree of a pbrt scene in motion using by adding values to the .objectMotion array. Here we'll set the Stanford bunny (labeled as "Default_B" in the asset tree) into motion with a vertical movement (middle term of the first triplet) and a tiny rotation (the second triplet). Later we'll show how the camera can also be set in motion:
% Move the bunny up, with slow rotation
if strcmp(sceneName, 'cornell box bunny chart')
pbrtCPScene.objectMotion = {{'Default_B', [0, 15, 0], [1, 1, 1]}};
Take Pictures using Auto, HDR, and Burst Intents
For the simplest case we ask our camera to take some pictures using pre-defined intents. By deault our camera assumes the 'Auto' intent. We also demonstrate 'Burst', which will capture more frames if the camera type we are using supports it. 'HDR' also fires a burst, but bracketed. Our default camera sums the frames in Burst mode, and chooses the highest non-saturated value at each pixel for HDR mode.
autoImage = ourCamera.TakePicture(pbrtCPScene, 'Auto',...
'imageName','Auto Mode');
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tpdac07f93_5bd6_44f0_827c_20d5ce4febc5\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 4.270669 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 49.1 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 24.1 sec ***
Docker container vistalab/pbrt-v3-spectral:latest
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
Docker command
docker run -ti --rm -w /Cornell_BoxBunnyChart -v C:/iset/iset3d/local/Cornell_BoxBunnyChart:/Cornell_BoxBunnyChart vistalab/pbrt-v3-spectral:latest pbrt --outfile renderings/Cornell_Box_Multiple_Cameras_Bunny_charts_depth.dat Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.1 sec ***
Reading image h=256 x w=256 x 31 spectral planes.
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tpfb3d5d1e_2e95_4921_8cb5_82526cad8d31\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.727483 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 43.4 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.7 sec ***
hdrImage = ourCamera.TakePicture(pbrtCPScene, 'HDR',...
'numHDRFrames', 3,'imageName','HDR Example', ...
'insensorIP', true);
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp77f0d7f8_6020_46b8_959c_5a0ef06d8c4a\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.418143 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 49.7 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 19.2 sec ***
Docker container vistalab/pbrt-v3-spectral:latest
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
Docker command
docker run -ti --rm -w /Cornell_BoxBunnyChart -v C:/iset/iset3d/local/Cornell_BoxBunnyChart:/Cornell_BoxBunnyChart vistalab/pbrt-v3-spectral:latest pbrt --outfile renderings/Cornell_Box_Multiple_Cameras_Bunny_charts_depth.dat Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.2 sec ***
Reading image h=256 x w=256 x 31 spectral planes.
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp39177c12_5e68_49cf_8f27_b4b0e5385910\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.543142 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 45.1 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 17.4 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp86078a23_9aef_4766_b1d1_a8430d27d6d4\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 2.214306 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 44.1 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.4 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp1c7f281b_5be6_40b5_8809_f74cc4b21ad6\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.684042 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 42.2 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 15.6 sec ***
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
hdrImage = ourCamera.TakePicture(pbrtCPScene, 'HDR',...
'numHDRFrames', 3,'imageName','HDR Example');
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp1577613a_e1a5_48c8_8698_694578dc11bc\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.951941 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 41.0 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 17.2 sec ***
Docker container vistalab/pbrt-v3-spectral:latest
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
Docker command
docker run -ti --rm -w /Cornell_BoxBunnyChart -v C:/iset/iset3d/local/Cornell_BoxBunnyChart:/Cornell_BoxBunnyChart vistalab/pbrt-v3-spectral:latest pbrt --outfile renderings/Cornell_Box_Multiple_Cameras_Bunny_charts_depth.dat Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.4 sec ***
Reading image h=256 x w=256 x 31 spectral planes.
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp5f69e2be_729d_42f6_bacd_c4f5fdc1b5d3\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.649317 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 45.1 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 19.7 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp051c8eda_c865_4d29_9b4c_ed4a6709d196\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.451097 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 44.9 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.8 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp8e1590ad_844f_4743_b721_b60febb11a24\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 2.159800 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 40.9 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 16.6 sec ***
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
burstImage = ourCamera.TakePicture(pbrtCPScene, 'Burst',...
'numBurstFrames', 3, 'imageName','Burst Example');
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp418d5fc6_7638_407e_8605_fbd68306807d\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 1.369447 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 45.5 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 21.0 sec ***
Docker container vistalab/pbrt-v3-spectral:latest
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
Docker command
docker run -ti --rm -w /Cornell_BoxBunnyChart -v C:/iset/iset3d/local/Cornell_BoxBunnyChart:/Cornell_BoxBunnyChart vistalab/pbrt-v3-spectral:latest pbrt --outfile renderings/Cornell_Box_Multiple_Cameras_Bunny_charts_depth.dat Cornell_Box_Multiple_Cameras_Bunny_charts_depth.pbrt
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 17.3 sec ***
Reading image h=256 x w=256 x 31 spectral planes.
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tp4e112a4f_b5a9_4298_9622_1dee6e9585bf\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 2.202332 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 44.5 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 17.9 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tpd8c8da2a_f568_4e18_ae4c_8f4688580a54\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 2.425751 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 44.3 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 18.8 sec ***
Overwriting PBRT file C:\iset\iset3d\local\Cornell_BoxBunnyChart\tpe061078d_db1f_46b7_91aa_8540e5c60513\Cornell_Box_Multiple_Cameras_Bunny_charts.pbrt
Elapsed time is 2.383180 seconds.
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts: 43.5 sec ***
*** Rendering time for Cornell_Box_Multiple_Cameras_Bunny_charts_depth: 18.0 sec ***
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Depth 75.98 Defocus -0.01 (1 of 8)
Depth 72.15 Defocus -0.01 (2 of 8)
Depth 68.32 Defocus -0.01 (3 of 8)
Depth 64.49 Defocus -0.02 (4 of 8)
Depth 60.65 Defocus -0.02 (5 of 8)
Depth 56.82 Defocus -0.02 (6 of 8)
Depth 52.99 Defocus -0.02 (7 of 8)
Depth 49.16 Defocus -0.02 (8 of 8)
Depth 75.98 Defocus 2.55 (1 of 8)
Depth 72.15 Defocus 2.55 (2 of 8)
Depth 68.32 Defocus 2.55 (3 of 8)
Depth 64.49 Defocus 2.55 (4 of 8)
Depth 60.65 Defocus 2.55 (5 of 8)
Depth 56.82 Defocus 2.55 (6 of 8)
Depth 52.99 Defocus 2.54 (7 of 8)
Depth 49.16 Defocus 2.54 (8 of 8)
Depth 75.98 Defocus 5.09 (1 of 8)
Depth 72.15 Defocus 5.09 (2 of 8)
Depth 68.32 Defocus 5.09 (3 of 8)
Depth 64.49 Defocus 5.09 (4 of 8)
Depth 60.65 Defocus 5.08 (5 of 8)
Depth 56.82 Defocus 5.08 (6 of 8)
Depth 52.99 Defocus 5.08 (7 of 8)
Depth 49.16 Defocus 5.08 (8 of 8)
Add Some Camera Motion
Note that we don't have a way to move the camera during frames (yet), so it only moves between frames. We also don't have a way to specify multiple cameras yet, so the first parameter is unused:
pbrtCPScene.cameraMotion = {{'unused', [1, 0, 0], [2, 2, 2]}};
% now see what the burst looks like with the camera rotating and
% translating slightly, perhaps mimicing a user's hand motion
%finalImage = ourCamera.TakePicture(pbrtCPScene, 'Burst', 'numBurstFrames', 3, 'imageName','Burst with Camera Motion');
Look at the changes we've made to our camera
cpCameraWindow(ourCamera, pbrtCPScene);
Working With ISET Scenes
While to get a fully-simulated 3D scene we need to use PBRT to render a Recipe, we can also take advantage of the variety of ISET Scenes (saved in .MAT files, or already loaded into ISET). Using them we can still emulate camera motion by moving the scene.
ourSceneFile = fullfile('Feng_Office-hdrs.mat');
isetCIScene = cpScene('iset scene files', 'isetSceneFileNames', ourSceneFile, ...
'sceneLuminance', sceneLuminance);
Reading multispectral data with mcCOEF.
Saved using svd method
% We can use the same Camera object if we want, but with this new scene.
autoISETImage = ourCamera.TakePicture(isetCIScene, 'Auto',...
'imageName','ISET Scene in Auto Mode');
% Set camera motion, which is simulated for iset scenes
isetCIScene.cameraMotion = {{'unused', [1, 0, 0], [2, 2, 2]}};
% now see what the burst looks like with the the camera (scene) moving
finalImage = ourCamera.TakePicture(isetCIScene, 'Burst', 'numBurstFrames', 3, 'imageName','ISET Burst with Camera Motion');
Working With Image Files
More sophisticated computational imaging algorithms typically rely on large data sets of training and test images. Neither of those exist in pbrt or iset format, so we also support standard RGB image files (.jpg, .png, etc.) We currently don't have a way to simulate object motion for these files, but we do support camera motion through simulating by moving the scene. Here we show an image using default rendering, and then with some camera motion and a burst of images. Note the color artifacts, since the default camera adds photosite values together in the raw sensor, but since the image has shifted, colors aren't correctly aligned:
ourImageFile = 'eagle.jpg';
imageCIScene = cpScene('images', 'imageFileNames', ourImageFile);
% We can use the same cpCamera if we want
autoImageCapture = ourCamera.TakePicture(imageCIScene, 'Auto',...
'imageName','Image File Captured in Auto Mode');
imtool(autoImageCapture);
% Set camera motion, which is simulated for iset scenes
imageCIScene.cameraMotion = {{'unused', [1, 0, 0], [2, 2, 2]}};
% now see what the burst looks like with the camera (scene) moving
finalImage = ourCamera.TakePicture(imageCIScene, 'Burst', 'numBurstFrames', 3, 'imageName','Burst from Image File with Camera Motion');
Using Pre-computed Scene and Object Motion
If we have access to a series of frames that illustrate either camera or object motion, we want to be able to use them directly. A simple example might be a series of frames from a video. To do that we can pass them as an array, either of ISET scenes or simply as image files
burstImageFiles = {'Crane_01.jpg', 'Crane_02.jpg', 'Crane_03.jpg'};
burstImagesCIScene = cpScene('images', 'imageFileNames', burstImageFiles, ...
'resolution',[256, 256]);
% We can use the same cpCamera if we want
autoImagesCapture = ourCamera.TakePicture(burstImagesCIScene, 'Auto',...
'imageName','Sequence of Image Files Captured in Auto Mode');
imtool(autoImagesCapture);
% Note that we can't add our own camera or object motion here, as we are
% simply using pre-computed scenes. Potentially we might want to support
% additional camera motion options, though?
% now see what the burst looks like with the camera (scene) moving
burstImage = ourCamera.TakePicture(burstImagesCIScene, 'Burst', 'numBurstFrames', 3, 'imageName','Burst from Image File with Camera Motion');
Generate a bracketed sequence from an image
Not sure how useful this is, if the original image is already LDR. But if we get some .exr scenes, we could experiment with those.
hdrBaseFile = {'BanteaySrei_01.jpg'};
hdrImageCIScene = cpScene('images', 'imageFileNames', hdrBaseFile, ...
'resolution',[256, 256]);
% We can use the same cpCamera if we want
hdrImageCapture = ourCamera.TakePicture(hdrImageCIScene, 'HDR',...
'imageName','Sequence of Image Files Captured in HDR Mode');
Using previously-captured Bracketed (HDR) Photos
We need to decide what to do in this case. The scene has already been captured with specific parameters that differ per frame, so re-capturing gets a bit tricky. Do we use the same exposure times but then "trick" our HDR algorithm by replacing those with the ones from the original photos? Or do something else?
% hdrImageFiles = {'BanteaySrei_01.jpg', 'BanteaySrei_02.jpg', 'BanteaySrei_03.jpg'};
% hdrImagesCIScene = cpScene('images', 'imageFileNames', hdrImageFiles, ...
% 'resolution',[256, 256]);
% % We can use the same cpCamera if we want
% hdrImagesCapture = ourCamera.TakePicture(hdrImagesCIScene, 'HDR',...
% 'imageName','Sequence of Image Files Captured in HDR Mode');
% imtool(hdrImagesCapture);
Test Focus Stacking code path
This doesn't work yet, as we have weird sizing issues with lens files, and trying to use depth for the current bunny scene isn't working because sizes are wrong. Need to use the new bunny asset once we figure out how, and have Brian walk us through lens files.
%stackedImage = ourCamera.TakePicture(pbrtCIScene, 'FocusStack',...
% 'imageName','Pretend Stack');
Print out some timing stats
Just for human consumption
tTotal = toc(getpref('ISET','tStart'));
beforeTime = getpref('ISET', 'benchmarkstart', 0);
disp(strcat("cpCam ran on: ", glData.Vendor, " ", glData.Renderer, "with driver version: ", glData.Version));
cpCam ran on: NVIDIA Corporation Quadro T2000/PCIe/SSE2with driver version: 4.6.0 NVIDIA 472.12
disp(strcat("cpCam ran in: ", string(afterTime - beforeTime), " seconds of CPU time."));
cpCam ran in: 1308.9219 seconds of CPU time.
disp(strcat("cpCam ran in: ", string(tTotal), " total seconds."));
cpCam ran in: 1313.6455 total seconds.
